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The meta-analysis study is a type of systematic review with strong scientific rigor; it has a 
number of characteristics that makes it a very usefiil tool. However, performing and reading 
meta-analysis could be a challenge — the meta-analysis overcomes the limitation of small sample 
sizes or rare outcomes by pooling results from individual studies in order to generate a single 
and better estimate. It also increases statistical power and allows the evaluation of discrepancies 
among the results of different studies. In this paper, we will present examples to illustrate how 
psychiatrists can utilize a meta-analysis in clinical and experimental research. 
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Scientific publications have shown a remarkable 
growth in the last years. However, among these pub- 
lications it is common to find studies with the same aims 
but with results that can be almost homogeneous or con- 
tradictory (1). These discrepancies may be due to various 
limitations in the studies such as the possibility of biases 
derived from original studies, the establishment of inclu- 
sion criteria, lack of quality of the studies or incorrect inter- 
pretation of the outcomes (2). As a consequence, it is 
necessary to perform analyses that allow the integration of 
evidences to clarify the results. Nowadays, meta-analysis 
has become a usefiil tool for medicine, epidemiology, social 
studies, ecology, medical engineering and other areas in 
which a synthesis of the scientific evidence is required (3). 

Objectives 

In the present study, we present a typical example of an 
association between genotype frequency and suicidal 
behavior to illustrate how psychiatrists may utilize a 
meta-analysis in clinical and experimental research. Sec- 
ond, we describe some generalities on the process to be 
followed in a meta-analysis and the main points to be 
considered when performing one. 

Definition 

The term meta-analysis was first introduced in 1976 and 
designated "any statistical analysis of a large collection 
of literature in order to integrate results" (1). 



Meta-analysis is a tool that has as objective the syn- 
thesis of published scientific evidence to obtain a better 
integration of the results. This analysis has been very 
useful in clinical and medical areas, since it provides 
several benefits (4). First, meta-analysis allows a further 
generalization of the results than those obtained from 
individual studies; this more general validation is a con- 
sequence of the fact that the sample does not come from 
a single population. In addition, this methodology 
increases the statistical power, and in turn increases the 
ability to find significant differences, as well as allowing 
more precision in the effect estimation. The meta-analysis 
study enables the measurement of discrepancies among 
the results of different studies and also provides possible 
explanations for this heterogeneity (5). 

The updated revisions of Cochrane Database of Sys- 
tematic Reviews facilitate the performance of meta- 
analysis in experimental studies that combine results or 
randomized clinical traits to ensure medical treatments 
(6). There are other statements for conducting revisions 
of this kind of studies such as the Preferred Reporting 
Items for Systematic Reviews and Meta-analyses, better 
known as PRISMA. This statement was developed using 
a consensus process oriented and updated by evidence; 
PRISMA consists of a 27-item checklist and a four-phase 
flow diagram. Its original version was proposed by 
QUOROM (Quality Of Reporting Of Meta-analysis). 
PRISMA is an essential tool for summarizing accurate 
and reliable evidence (7). 
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Aims 

The first objective of meta-analysis is to obtain clear and 
reliable results useful in the management of patients and 
possibly as a basis for clinical guidelines (1). Meta- 
analysis, when is used correctly, must comply with the 
following objectives: 1) to test the hypothesis related to 
the effect of the intervention under study; 2) to increase 
the accuracy of the estimators of the effect of the inter- 
vention under study; 3) to assess the consistency between 
clinical trials of similar interventions associated with the 
topic and generate a more efficient estimator of the effect; 
4) to assess the consistency between trials of different 
interventions performed for the same purpose and gener- 
ate an estimate of the effect of such care; 5) to identify 
with accuracy subgroups of patients who would most 
likely be affected by the intervention, either in a favor- 
able or unfavorable manner, and 6) to calculate the 
requirements, in terms of sample size, of future clinical 
trials to be performed in the same field (8). 

The meta-analysis study contributes not only its ver- 
satility but also its pertinence in several aspects of clini- 
cal research. This type of research study increases the 
statistical power of comparison and also improves the 
estimation effects (9). Several issues can be evaluated 
with a meta-analysis; it is particularly useful in studies 
with contrasting results when one wants to combine 
these results, or when it is necessary to analyze sub- 
groups of subjects selected from different studies, or in 
the searching for answers to new questions (5). 

Design of a meta-analysis study 

Stating the problem 

The first phase in the meta-analysis is stating the prob- 
lem. The researcher has to formulate the problem of 
interest, otherwise the lack of clear and precise answers 
and the possible factors can become a matter of confu- 
sion and lead to biased results (6). In addition, the extrac- 
tion of specific data from the studies is a relevant process, 
as well as the selection of the most appropriate statistical 
techniques for the analysis. Also, if there are previous 
meta-analyses of the relationship studied, the researcher 
must clarify the possible differences between them. A 
correct definition and delimitation of these aspects facili- 
tates the next stages of the process of meta-analysis (4). 

Example 1. The following text comes from a previous 
meta-analysis where we evaluated the association of sui- 
cidal behavior and 5-HTR2A gene (8). 

Suicidal behavior is a major health problem worldwide. 
Several recent studies have been carried out that support 
a possible relationship between genetic factors and sui- 
cidal behavior (10). Historically, evidence for the involve- 
ment of serotonin (5-HT) in suicide originated from 
findings of low 5-hydroxyindoleacetic acid concentration 



(5-HIIA) in cerebrospinal fluid (CSF) of depressed sui- 
cide attempters and in brain stems of completed suicides 
(11). These studies provided evidence for altered seroton- 
ergic neural transmission in the pathogenesis of suicidal 
behavior. In consequence, genes pertaining to the sero- 
tonergic system have been proposed as candidates to 
establish biological correlates between suicidal behavior 
and the serotonergic system. One candidate gene in the 
study of suicidal behavior is the gene encoding for the 
serotonin 2A receptor (12). In this example, we can 
appreciate the delimitation of a topic for study. 

Effect size 

The term effect size is an index used to quantify the 
relationship between two variables or a difference 
between two groups. This measure is based on means, 
binary data or correlations (5). 

Effect size based on means 

This type of index is employed when the studies report 
means and standard deviations. The preferred effect size 
is usually the raw mean difference, the standardized mean 
difference, or the response ratio (6). Also, Cohen's is an 
example of standardized mean difference in the effects 
sizes based on means (13). 

Effect size based on binary data 
This measure includes data from prospective studies, such 
as a randomized trial, which is originally reported as the 
number of events and non-events in two groups. Research- 
ers typically compute a risk ratio, and/or risk difference 
(14, 15). 

Effect size based on correlations 
This index is used in studies that report a correlation 
between two continuous variables; the correlation coeffi- 
cient itself could serve as the effect size index. The cor- 
relation is an intuitive measure that has been standardized 
to take into account different metrics in the original 
scales (5). 

Example 2. The following example was extracted from a 
previous meta-analysis where we evaluated the associa- 
tion of suicidal behavior and the COMT gene (14). 

Table 1 presents the results of a meta-analysis of case- 
control studies on the role of the COMT (catechol- 
O-methyltrasferase) vall58/108Met polymorphism in 
suicidal behavior. 

Searching of information 

In general, the information of interest in the studies 
included in a meta-analysis must be extracted taking into 
consideration the following aspects (7, 16). 
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Table 1. Meta-analysis of case-control studies on the role of the COMT (catechol-O-methyltrasferase) 
vall58/108Met polymorphism in suicidal behavior. 



Number of COMTval Number of COMTmet 

alleles alleles 



References 


Cases 


Controls 


Cases 


Controls 


Odds ratio (95% CI) 


Tovilla-Zarate (2011) 


126 


272 


84 


200 


0.90 (0.65-1.26) 


Lee (2011) 


223 


273 


117 


121 


1.18 (0.86-1.61) 


Perround (2010) 


848 


255 


784 


221 


1.06 (0.86-1.30) 


Zalsman (2008) 


182 


121 


220 


117 


1.25 (0.90-1.72) 


Rujescu (2003) 


139 


323 


159 


333 


1.10 (0.84-1.95) 


Liou (2001) 


95 


275 


29 


101 


0.83 (0.51-1.33) 


Russ (2000) 


52 


58 


46 


40 


1.28 (0.72-2.25) 


Ohara (1998) 


11 


175 


13 


95 


2.17 (0.93-5.04) 


Random effects 


1676 


1752 


1452 


1228 


1.09 (0.97-1.23) 



Characteristics of the study 

These include type of design, description of the sample 
(age, gender, diagnosis, among others), type of interven- 
tion (dose, active ingredient, just to mention a few), 
follow-up time of an evaluation and other features that 
may help in assessing the homogeneity or heterogeneity 
in a pool of results of the studies included. 

Quality of the methodological study 

To evaluate this aspect it is important to use instruments 

that can detect the possibility of bias. 

Results 

At this point, measures of the observed effect (odds ratio, 
relative risk, significant difference, among others) are 
provided with indicators of variability (confidence inter- 
vals) and statistical significance. 

Localization of the research studies 

The quality of a meta-analysis depends on the type of 
search performed to identify and locate the original 
papers (16). 

A literature search is made of informal, primary and 
secondary sources. Informal sources comprise books, 
personal files, review articles, among others (6). Primary 
sources are known journals related to the topic, as well 
as complementary revisions consisting of further selec- 
tion of articles cited in primary sources. Also, secondary 
sources are automatized databases; among the most 
important are MEDLINE, EMBASE, Web of Science 
and Cochrane databases (17). In addition, is important to 
note that the selection of the databases depends on the 
topic, because there are many specific databases. 

Example 3. The following example was taken from a pre- 
vious meta-analysis where we evaluated the association 
between suicidal behavior and the 5-HTRlA gene (18). 



Identification and selection of publications 
A literature search was performed covering from April to 
June 2010. Relevant publications were identified using 
the following search terms in Medline, PubMed and Web 
of Science databases: "HTRIA and suicidal behavior", 
"HTRIA and suicide", "rs6295 and suicidal behavior", 
"rs6295 and suicide", and "HTRIA C-1019G and sui- 
cide". These words were combined to retrieve the sum- 
maries. The search also implicated the review of the 
bibliography cited at the end of various research articles 
(18). For more examples, see (19). 

Inclusion and exclusion criteria of the studies 

Not all the papers localized in the search may be 
included in the meta-analysis. The researchers should 
comply with the requirements to consider their inclu- 
sion in the meta-analysis. At this step of the meta- 
analysis one may incur in a "selection bias". To reduce 
the risk of committing bias, the review of the studies 
should be conducted by different researchers following 
a list of inclusion and exclusion criteria. In this man- 
ner, the reliability and accuracy of the meta-analysis is 
maximized (5). 

Evaluation of the quality of the 
studies included 

In every meta-analysis it is important to assess the qual- 
ity of the studies included and this implies to perform a 
control study. There are some basic aspects that must be 
controlled in the information that is being acquired such 
as study design, the possibility of combining different 
studies, bias controls and statistical analyses for each 
study included in the meta-analysis. The Newcastle- 
Ottawa Scale (NOS) was developed to assess the quality 
of non-randomized studies. The value of this instrument 
lies in its design, content and ease of use directed to the 
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task of incorporating quality assessments in the interpre- 
tation of the results of a meta-analysis (20). 

Example 4. The following example was drawn from a pre- 
vious meta-analysis where we evaluated the association of 
suicidal behavior and the 5-HTR2A gene (8). 

Statistical analysis 

Studies deemed for inclusion in the systematic review 
were scored for methodological quality using the New- 
castle-Ottawa Assessment Scale. A score of six was 
taken as the cut-off point to distinguish higher from 
lower quality studies. Quality assessment was done by 
the same two authors (TBG-C, CAT-Z) based on the 
NOS instrument. 

Analysis of heterogeneity 

There are several statistical and graphical methods to 
evaluate the heterogeneity of the meta-analysis, but in 
general all statistical tests designed to verify the existence 
of heterogeneity are based on the assumption of zero 
variability among studies. One of the most appropriate 
tests to evaluate heterogeneity is the Q test proposed by 
Dersimonian and Laird, preferred for issues of validity 
and computational simplicity. The Q test can be supple- 
mented with some graphical representation to allow a 
visual analysis of the variability among studies. The most 
used representations are the Galbraith graph — recom- 
mended in observational and experimental studies — and 
the L'Abbe plot, which is more restrictive and is only 
applicable in meta-analysis of clinical trials (5). 

The Galbraith graph exhibits the precision of each 
study and is calculated as the inverse of the standard 
error versus the standardized effect; it also represents the 
fitted regression line to these points and a confidence 
band. Studies that fall outside this band are the main 
contributors of heterogeneity. In addition, the position of 
the studies on the abscissa (x-axis) allows a visual iden- 
tification of those studies with more weight in the meta- 
analysis. The Galbraith graph can also be used to detect 
additional sources of heterogeneity in labeling studies as 
well as different variables, such as the year of publica- 
tion (3). On the other hand, the L'Abbe plot is more 
restrictive than the Galbraith graph. It is only applicable 
in meta-analysis of clinical trials. Moreover, this type of 
graph can only be built when the effect size is based on 
binary data (7). 

Many statistical tests are available for evaluating hetero- 
geneity between studies. Higgins and colleagues, in two 
highly cited papers (21, 22), proposed the routine use of 
the 12 statistic. This statistic can be used to compare the 
amount of inconsistency across different meta-analyses 
even with different numbers of studies. 12 is routinely 
implemented in all Cochrane reviews and is increasingly 
used in meta-analyses published in medical journals (23). 



To explain heterogeneity requires experience. When 
heterogeneity is encountered in a meta-analysis, there are 
some options available. The researcher may provide a 
summary measure, even when heterogeneity is present or 
may not proceed with a summary of the primary studies 
(6). If the researcher decides to analyze the results 
despite heterogeneity, it is necessary to measure the vari- 
ability "among studies", "intra-studies" and the variation 
coefficient among studies. If the researcher suspects that 
there are reasons to explain the heterogeneity of results 
among studies, the most recommended option is to per- 
form a subgroup analysis combining only those studies 
that meet a certain condition or feature, to get a more 
homogeneous sample (5). 

Example 5. The following example comes from a previous 
meta-analysis in which we evaluated the association of 
suicidal behavior and the COMT gene (14). 

The pooled OR derived from all studies indicated a non- 
significant association of the met allele in the COMTval/ 
met polymorphism with suicidal behavior (Random 
effects model: 0R= 1.07; 95% CI 0.85-1.33; P(Z) = 0.19). 
We observed heterogeneity in all studies {Q = 57.08, 
df= 1; = 0.0005). Subsequently, we carried out a sec- 
ond analysis, which only included studies inside the het- 
erogeneity curve. As a result, we could not find an 
association either (OR= 1.09, 95% CI 0.97-1.23; 
Z= 1.11, P(Z) = 0.26). 

Example 6. The following example was taken from a pre- 
vious meta-analysis where we evaluated the association 
between suicidal behavior and the 5-HTRlA gene (18). 

The pooled OR derived from all studies indicated a non- 
significant association of the G allele of rs6295 and sui- 
cidal behavior (Random effects model: OR =1.08; 95% 
CI = 0.80-1.45; P{Z) = 0.80)- Heterogeneity was observed 
in all studies ((?= 17.84; df=4; /* = 0.0013). Subse- 
quently, we performed a second analysis, which only 
included studies inside the heterogeneity curve (Italian, 
German, Ukrainian and Korean samples) (18). 

Combination of results 

There are several statistical techniques for combining and 
presenting the results in a meta-analysis study. The choice 
of the method depends on the type of the outcome/effect 
used and on assessing the degree of heterogeneity of the 
study results. Most meta-analyses are based on one of 
two statistical models: the fixed effects model and the 
random effects model (7). 

Fixed effects model 

Under the fixed effects model, it is assumed that all stud- 
ies in the meta-analysis share a common effect size, that 
is, all factors that may influence the effect size are the 
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same in all the studies. It makes sense to use the fixed 
effects model if two conditions are met. First, the idea 
that all studies included in the meta-analysis are function- 
ally identical. Second, when the goal is to compute the 
common effect size for the identified population, and not 
to try and make a generalization to other populations (2). 

Random effects model 

This model assumes that the studies have enough in com- 
mon that it makes sense to synthesize their information, 
but there is in general no reason to believe that they are 
all identical in the sense that the common effect size is 
exactly the same in all the studies. When the researcher 
is accumulating data from a series of studies performed 
by researchers operating independently, it is unlikely that 
all the studies will be functionally equivalent (6). Typi- 
cally, the subjects or interventions in these studies may 
differ in ways that can have an impact in the results. 
Therefore, we must not assume a common effect size. In 
these cases, the random effects model is more appropri- 
ate than the fixed effects model (2, 3). 

Interpretation of results 

The evaluation of the size of the pooled effect, the pos- 
sible causes of heterogeneity and the evaluation of the 
stability of the meta-analysis are involved in the interpre- 
tation of the results (5, 18). 

Graphical representation of results 

The usual way for displaying data from a meta-analysis 
is a pictorial representation. The forest plot provides con- 
text for the analysis. The plot highlights the effect sizes, 
the sum of effects, confidence interval, and odds ratio 
or risk ratio. To this end, when building a graph, the 
abscissa axis (x-axis) depicts the viewed effect (odds 
ratio, relative risk, among others) and on the coordinate 
axis (y-axis) lie the different studies, usually ordered by 
year of publication or some other arrangement (3, 7). 

Example 7. The following example was extracted from 
a meta-analysis where we evaluated the association of 
suicidal behavior and the 5-HTR2A gene (8). 

Figure 1 shows only the evaluation of C allele vs T 
allele. Odds ratios and forest plots of all models in over- 
all studies. 

Limitations 

The meta-analysis technique presents certain limitations 
in methodology. These limitations must be recognized 
and taken into account when interpreting the results. 
First, the meta-analysis can lead to distorted results 
due to possible bias in the selection and publication of 



studies. In addition, the validity of the results and con- 
clusions of the meta-analysis depend on the quality of 
the individual studies, so the combination of biased stud- 
ies can further enhance the bias. Finally, the interpreta- 
tion of meta-analysis for heterogeneity or variability 
between studies is difficult and controversial. The com- 
promise of those who use the meta-analysis technique is 
to understand these limitations and discuss them explic- 
itly and in each case. Below we describe some of the 
main methodological problems of this approach (7, 15). 

Heterogeneity among studies 

Some variation in the results of the studies is expected 
due to chance alone, but an excess of variability reflects 
true differences in the results of the trials; this situa- 
tion is called "heterogeneity". A first methodological 
criticism of meta-analysis is to attempt a statistical 
combination of results from studies that show great 
variability among them. This difficulty is not unique to 
meta-analysis, since it is shared by all clinical research. 
The wide variety of features inherent in the subjects of 
the study makes it necessary to design a uniform proto- 
col, conduct a rigorous process of selection of the par- 
ticipants in the study and a subsequent careful analysis 
of the influence of the extreme cases or outliers in the 
results (6, 18). 

Publication bias 

While a meta-analysis yields a mathematically accurate 
synthesis of the studies included in the analysis, if these 
studies are a biased sample of all relevant studies, then 
the mean effect computed by the meta-analysis will 
reflect this bias. Several lines of evidence show that stud- 
ies that report relatively large effect sizes are more likely 
to be published than studies that report small effect sizes. 
Since published studies are more likely to find their way 
into a meta-analysis, any bias in the literature is likely to 
be reflected in the meta-analysis (18). 

Example 8. The following example was taken from a pre- 
vious meta-analysis in which we evaluated the association 
between suicidal behavior and the COMT gene (14). 

Our study presents some limitations. With regard to the 
meta-analysis, publication bias has to be considered, 
since negative studies are less likely to be published. 
Also, an overrepresentation of the results showing an 
association between the polymorphism and the investi- 
gated disorder is also possible (24). Although the con- 
tribution covering from genetic factors to personality 
traits may differ between male and female subjects, we 
did not analyze for gender. Other limitations are inher- 
ent in many meta-analysis of association (including this 
one) such as their retrospective nature and the inclu- 
sion of study-level data. 
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Study (Year) n 

Zhang (1997) 330 
Du (1999) 110 
Tsai (1999) 314 
Turecki (1999) 250 
Bondy(2000) 512 
Du (2000) 418 
Preuss (2000) 324 
Preuss (2000) 358 
Crawford (2000) 326 
Crawford (2000) 418 
Arias (2001) 394 
Chen (2001) 1674 
Tan (2002) 282 
Pooley (2003) 142 
Ertugrul (2003) 142 
Vadim (2005) 234 
Correa (2007) 258 
Zhang (2007) 1084 
Vaquero-Lorenzo (2008) 1702 
Wrzosek (2011) 277 
Tovilla (2012) 810 
Random effects 11133 




OR Cl(95.0%) 

2.63(1.16,5.93) 
0.88(0.39,1-98) 
1.16(0.73,1.86) 
5.40(3.11,9.34) 
0.96(0.67,1.36) 
1.81(1.21,2.72) 
1.15(0.70,1.88) 
0.95(0.61,1.48) 
0.64(0.41,1.01) 
1.81(1.21,2.72) 
1.31(0.76,2.28) 
0.87(0.70,1.08) 
1.04(0.62,1.77) 
0.90(0.67,1.21) 
1.25(0.63,2.47) 
1.21(0.72,2.02) 
0.86(0.51,1.46) 
0.91(0.71,1.15) 
1.22(1.00,1.47) 
0.89(0.49,1.62) 
0.68(0.50,0.94) 
1.14(0.96,1.35) 



0.6 0.7 0.8 0.9 1 2 3456789 

Fig. 1. Odds ratios and forest plots of all models in overall studies. C allele vs T allele. 
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Meta-analysis of GWAS 

In other hand, recently, in order to know the Genes of 
Major Effects in complex diseases as suicide behavior, 
bipolar disorder, schizophrenia and others psychiatric dis- 
orders, the meta-analysis of genome-wide association 
studies (GWASs) has become a popular method. Some 
researchers sometimes run a GWAS followed by a repli- 
cation study and then meta-analyses the replication data 
with the discovery data in order to capture the totality of 
the evidence. In this regard, we can observe two types of 
meta-analysis of GWAS: for discovery and for replica- 
tion. Within the meta-analysis for discovery, two designs 
are commonly found in the literature. In the first case, 
the members of a consortium have the opportunity to 
work together to ensure the comparability of their quality 
control and primary analyses, and to collaborate on more 
detailed follow-up analyses should interesting effects be 
observed. Probably, at the second design the researcher 
find difficult to obtain genome-wide significance, so go to 
the web in search of publicly available GWAS data that 
can be combined with their primary studies in order to 
obtain more precise results. For a review, see (25, 26). 

Conclusions 

Finally, as occurs with all research studies, a meta-analy- 
sis must draw conclusions; this part of the meta-analysis 
must be clear, not extensive, and must summarize the 
principal results of the meta-analysis. 

Conclusions of this article 

The meta-analysis study has become a helpful tool in 
psychiatry since it systematically summarizes available 
evidence. The meta-analysis allows more generalization 
in the results of individual studies, possesses higher 
validity because the samples come from different studies 
and are not restricted to the same population. Even 
though the meta-analysis study provides several benefits 
in clinical and research areas, it is rather important to 
understand each of its component parts. 
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